This disclosure relates generally to the field of digital photography and video capture. More particularly, but not by way of limitation, it relates to techniques for improved stabilization of capture video frames by intelligently harnessing the complementary effects of both optical image stabilization (OIS) and electronic image stabilization (EIS).
A standard rule of thumb for capturing sharp, handheld imagery is that the camera's shutter speed should not be less than its shutter speed equivalent to the focal length of the lens. This rule holds that a 500 millimeter (mm) lens shouldn't be handheld at shutter speeds slower than 1/500-second, a 300 mm lens slower than 1/300-second, a 50 mm lens slower than 1/50-second, and a 20 mm lens slower than 1/20-second. With the application of software- and/or hardware-based stabilization technology, jitter caused by camera movement may be minimized, making it possible to transform shaky, handheld footage into steady, smooth shots.
One way to stabilize a video is to track a salient feature in the image and use this as an anchor point to “cancel out” all perturbations relative to it. This approach requires a priori knowledge of the image's content to, for example, identify and track a person or other salient object in the scene. Another approach to image stabilization searches for a “background plane” in a video sequence, and uses its observed distortion to correct for camera motion. These types of techniques that involve software- and/or hardware-enabled transformations to “warp” a captured image so as to “cancel out,” i.e., stabilize, the jitter caused by camera movement after-the-fact will be referred to herein as “electronic image stabilization” (EIS) and/or “video image stabilization” (VIS) techniques. Another approach may use the output from a motion sensor, e.g., a gyroscope, as an input for estimating the amount of “warping” that needs to be done via the EIS transformations in order to stabilize the video frames.
In another approach, gyroscopically controlled electromagnets (or other suitable mechanism) shift a floating lens element orthogonally to the lens barrel axis (i.e., the “optical axis”) along the horizontal and vertical plane of the image and/or along the optical axis in a direction that is opposite that of the camera movement. Doing this can effectively neutralize any sign of camera shake. In a similar type of operation, a camera's imaging sensor may translated in the opposite direction of the camera's movements in order to dampen the effects of camera shake. These types of techniques that involve hardware-enabled corrections in the position of the image capture apparatus, e.g., by moving one or more elements in the optical stack, the image sensor itself, or the entire camera system, so as to “cancel out” the jitter caused by camera movement in “real-time” will be referred to herein as “optical image stabilization” (OIS) techniques.
One limitation of current EIS video stabilization techniques is that, because they primarily attempt to correct for frame-to-frame distortions between captured images, they do not do a good job of accounting for so-called “motion blur” occurring within images, e.g., due to the user's hand shaking during video capture. “Motion blur” may be defined as the apparent “streaking” or “blurring” of rapidly moving pixels in a still image or a sequence of images. Motion blur results when the composition of the image being recorded changes during the recording, i.e., exposure or integration, of a single frame, either due to rapid movement (of the camera or objects in the scene being captured) or long exposure times, i.e., “integration times.”
The difficulties associated with stabilizing video frames exhibiting motion blur are further exacerbated in low light conditions due to the increased integration times needed to capture sufficient light in the recorded video frames. Longer integration times result in more motion blur in the recorded video frame. When such “low light” video is stabilized, the residual motion blur and associated “shimmering artifacts” often appear visible in the stabilized video. This may make the stabilized videos look unnatural and does not provide the user with the perceptual clues of video movement that would normally be associated with the presence of motion blur within a video frame.
Due to its ability to move one or more elements in the optical stack as the image is being captured, OIS can correct or reduce motion blur during capture. However, due to physical size limitations within most portable electronic devices, the OIS techniques do not provide enough correction range for typical video capture. In addition, OIS techniques can only correct for image translation, not affine/perspective correction, which can be done through “EIS” techniques.
Thus, what is needed are techniques to intelligently modulate the strengths—and types, i.e., EIS and OIS—of image stabilization techniques used to stabilize particular recorded video frames in a sequence of captured frames based, at least in part, on: 1.) estimated motion data recorded at the image capture device during capture; 2.) estimated lens position during capture; 3.) estimated light levels during capture; and/or 4.) the frequency band composition of the motion data recorded at the image capture device during capture. Such techniques are also preferably power efficient and computationally efficient.